Planning for Operational Control Systems with Predictable Exogenous Events

Various operational control systems (OCS) are naturally modeled as Markov Decision Processes. OCS often enjoy access to predictions of future events that have substantial impact on their operations. For example, reliable forecasts of extreme weather conditions are widely available, and such events can affect typical request patterns for customer response management systems, the flight and service time of airplanes, or the supply and demand patterns for electricity. The space of exogenous events impacting OCS can be very large, prohibiting their modeling within the MDP; moreover, for many of these exogenous events there is no useful predictive, probabilistic model. Realtime predictions, however, possibly with a short lead-time, are often available. In this work we motivate a model which combines offline MDP infinite horizon planning with realtime adjustments given specific predictions of future exogenous events, and suggest a framework in which such predictions are captured and trigger real-time planning problems. We propose a number of variants of existing MDP solution algorithms, adapted to this context, and evaluate them empirically. Introduction Operational control systems1 (OCS) are devised to monitor and control various enterprise-level processes, with the goal of maximizing enterprise-specific performance metrics. These days, OCS is a beating heart of various heavy industries, financial institutions, public service providers, call centers, etc. (Etzion and Niblett 2010). While different OCS employ different degrees of automation, a closer introspection reveals an interesting wide common ground. In most cases, as long as the enterprise is operating in its “normal conditions”, its control by the OCS is mostly automatic, carried out by a default policy of action. However, once deviations from the “normal conditions”, or anomalies, are either detected or predicted, it is common that a human decision maker takes control of the system and changes the policy to accommodate the new situation. Anomalies, such as machine failures, severe weather conditions, and epidemic ∗Brafman and Domshlak were partly supported by ISF Grant 1101/07. We thank anonymous reviewers for helpful comments. Copyright c © 2011, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. We note that the terminology in this area is still evolving, and OCS are called differently in different domains. bursts are detected either by system operators, or through an automatic event-processing system. Because the state models of enterprises adopting OCS are complex and because the decisions need to be timely, the outcome of human decisions in face of exogenous events are often sub-optimal, leading to unnecessary high costs for the enterprises. Achieving a higher degree of OCS automation is thus clearly valuable, and this is precisely the agenda of our work here. At a high level, OCS are naturally modeled as Markov decision processes (MDP), and previous works on MDPs with exogenous events suggested compiling a prior distribution over these events into the probabilistic model of effects of actions performed by the system (Boutilier, Dean, and Hanks 1999). OCS, however, challenge this approach threefold. First, the space of exogenous events and their combinations, which may impact an OCS can be very large, prohibiting their modeling within the MDP. Second, for many of these exogenous events there is no useful predictive, probabilistic model. Finally, even if accurate priors are available, realtime predictions, possibly with short lead-time, make these priors effectively useless. Given an information about ongoing and near-coming events, the model has to be updated, and a good/optimal policy for it needs to be computed. The major computational issue here is that the new policy should be computed in real time, during system operation, and thus there is usually not enough time to solve the new model from scratch. In this work we introduce a model that combines offline MDP infinite horizon planning with realtime adjustments given specific predictions of exogenous events, and suggest a concrete framework in which such predictions are captured and trigger real-time planning problems. At a high level, the model is based on the characteristic OCS property, whereby exogenous events constitute anomalies, derailing the system from some concrete default mode of operation. Specifically, the model exploits the fact that (i) the MDP representing the system is expected to differ from the default MDP only within a specific time interval affected by a set of exogenous events, and that (ii) most of the actions prescribed by the pre-computed policy for the default mode of operation are likely to remain optimal within the event-affected period. Under this perspective, we suggest modeling predictions of exogenous events as trajectories in a space of different MDPs, each capturing system operation under speProceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence